fix(search)!: fan out a qualified edge into one entry per tuple - #801
Open
ddeboer wants to merge 8 commits into
Open
fix(search)!: fan out a qualified edge into one entry per tuple#801ddeboer wants to merge 8 commits into
ddeboer wants to merge 8 commits into
Conversation
A weld asks whether ONE entry satisfies every condition. An entry whose leaves hold arrays stands for every combination at once, so it answers the weld with none of them – and Typesense 30.2, the current stable release, does not answer at all: the search hangs indefinitely, which reaches a consumer as an unexplained error. - refuse a nested field declaring both filterable and array, so a leaf a weld can name states one value per entry - fan an edge out into one entry per combination of its weldable leaves, on the framed node before projection, so each leaf still passes through transform, folding and the facet companion unchanged - write a nested identity companion under the arity of its reference, since that companion is the leaf a weld actually names; a top-level companion stands for the whole document and is unchanged - bound the fan-out with maxEntries (default 100), per ADR 12: a cartesian product over an edge's values is a bound in the data's own units - pin the engine guarantee against a real container, the array-shaped hang included, so a future engine bump reports when it is gone Measured at 1,000,000 documents: the welded filter answers in 14–19 ms where it previously never returned, and the fanned-out shape is no slower than the array shape on 31.0, so the eventual upgrade unwinds nothing. Indexing is ~15 % slower for ~50 % more entries. BREAKING CHANGE: a nested field may no longer declare both “filterable” and “array”; searchSchema refuses it. Declare the leaf single-valued – the projection now emits one entry per combination.
Three defects found reviewing the fan-out, two of them introduced by it and each invisible without a live engine. - restore the already-a-list case when widening a nested leaf. Only Reference Types are held to the single-valued rule, and the same path declares the fields of the Root Type a local lookup nests, where array and filterable meet on an ordinary facet – the leaf was emitted with no type at all, which the engine refuses outright - declare a nested identity companion by what its path yields rather than always as a list. Under a single-valued edge the parent is an object, nothing flattens, and Typesense rejects the scalar the projection writes against a declared list: the whole import fails - expand every tuple the cap keeps. Stopping mid-expansion left the remaining weldable leaves holding their lists, and a single-valued leaf then kept the first value and dropped the rest – losing data quietly rather than fanning it out Two existing tests asserted the arity rule inverted, stating that a companion is always a list and that a single value is what the engine rejects. A live import says the opposite, so both now read the other way. An identity on a Reference Type's field is already refused, and a nested filterable lookup can no longer be an array, so a nested companion never holds more than one id; no change was needed there. The entry budget is per node carrying the reference rather than per document, so nested edges compound it – documented rather than changed.
…e arity 30.2 will not weld Two limitations found verifying the fan-out against a live engine, neither visible from a collection definition alone. - a nested identity companion is typed by the path that reaches it rather than by what one entry holds, and Typesense enforces that wherever nothing widens the path - the weld hangs on 30.2 against a single-valued edge whatever its leaves hold, which is a different defect from the array one and is not addressed by fanning out. It costs nothing today, because a qualified relation is multi-valued by nature
…its type A Root Type reached by a local lookup is nested exactly as a Reference Type is; it just happens to have a collection of its own elsewhere. Reading that from the type made the projection treat a locally-nested root as a root, so it wrote a list where the collection declared a single value – and Typesense rejects the document at import, for every document carrying such a reference. - thread the nesting flag from the call site into projectFields, rather than deriving it from whether the declaring type declares a class - declare a nested identity companion by both routes to a list, the field's own array as well as a flattening ancestor, as every sibling declaration in that function already does - reject a maxEntries that is not a positive integer. The budget is counted off one entry at a time, so a fractional or non-positive cap is never reached and the fan-out grows with the data – the unbounded case the cap exists to prevent. Compare with >= rather than ==, so no cap can be stepped over - export DEFAULT_MAX_ENTRIES beside DEFAULT_LABEL_FIELD, and state the number in the option's own documentation, so the default is legible without it Left alone deliberately: a single-valued inline reference still fans out up to the full budget before keeping one entry. Capping it at one would change which entry survives when the first projects empty, and the waste is bounded.
…hat survives Three ways the entry budget took data it was never meant to touch, each verified against the projection before and after. - cap only a reference that CAN fan out. The budget bounds the multiplication this decision introduces, but it was applied to every inline reference: a display-only nesting of 300 entries silently became 100, shortening a list that has always been stored whole - charge it against the entries that SURVIVE, not the framed values considered. A run of values the reference type reads nothing from – dirty source data, which the emptiness filter exists for – spent the budget and left the real referents behind them with nowhere to go, dropping the reference entirely - narrow a nested companion only where fan-out underwrites it. An identity is earned by filterable OR facetable, and only a weldable leaf is ever split, so a facetable-only companion kept its first id and dropped the rest – silently, because the collection declared it a single value too The declared type follows the same rule, so the two agree again. Also drop an unreachable guard: tuplesOf is handed what remains of the budget and never returns more, so the inner loop cannot overshoot it. The ADR claimed entries past the cap are “dropped and reported”. There is no reporting channel and inventing one for data that should never arise is not worth the seam, so it now says the drop is silent, that truncation follows declaration order rather than being representative, and why a local lookup is uncapped.
The fan-out needs a bound – a cartesian product over an edge's own values is no bound at all – but capping the entries a document stores is not it. By the time the projection runs every cost has been paid: the CONSTRUCT matched those values and the endpoint paid for it, they crossed the wire, the subject index holds them, and framing has materialised them into one node. Capping the product declines the last and cheapest step while keeping all the expensive ones, and bounds nothing framing did not already hold. It also missed the case that predates fan-out entirely: a wide edge, or a display-only nesting of ten thousand entries, is linear rather than multiplicative and was never bounded either. - remove maxEntries, its default, its validation and its export - remove the budget from the projection, and with it the survivor-charging and part-way expansion the cap alone made necessary - record in ADR 26 why the bound belongs at the framing seam instead, and that the fan-out is unbounded until it lands The bound is tracked separately, where it can be decided on its own evidence: capping values per leaf where they enter memory makes the product k^(weldable leaves), and leaf count is a schema constant – a bound in the schema's own units rather than a number written against the data's. Lower the branch threshold by 0.01 to match: the tests that raised it existed only to exercise the cap, and the code they covered has gone with it.
ddeboer
force-pushed
the
worktree-weld-fanout
branch
from
September 4, 2026 12:31
b57bd33 to
2edf409
Compare
…sition Nothing welds the identity field by name, so fan-out passed over it – but the flat companion harvested FROM it is the leaf a weld uses to name the endpoint. An entry whose identity field held three ids therefore stood for three endpoints at once, and the companion kept the first and dropped the rest: a filter on either of the others missed a document that genuinely carried it, with the entry beside it still listing all three. The same tuple problem the fan-out exists to remove, one field further in, so it takes the same answer: the identity field is a tuple position like any weldable leaf, and multiplicity moves to the entry list. Three endpoints now make three entries holding one endpoint each, and every id stays reachable. This is what the narrowing in setIdentity always assumed. It justified itself with "fan-out splits weldable leaves only", which was true of `filterable` fields and not of the one the companion actually reads.
… drops Fan-out splits RAW framed values, and two distinct ones can still collapse downstream – a transform mapping two spellings onto one canonical value, or a keyed target re-keying two referent IRIs to the same document key. Before fan-out those values met inside one entry and were deduped there; split across entries they reached the index and the API as byte-identical duplicates. - dedupe the projected entries by content, restoring the guarantee the pre-fan-out shape had - document what a single-valued edge now drops. It stores one entry, so an edge the graph gave two endpoints indexes the first where the old single entry listed both. The old shape was never answerable by a weld, so this is the ordinary single-valued rule meeting data that outgrew the declaration – but it changes what a filter matches, and nothing can refuse it at startup because the declaration is only wrong once the data has more than one value - correct the justification shared by setIdentity and nestedIdentityFields. Both claimed a facetable-only identity is never split, which stopped being true when the identity field became a tuple position: it is always split, and `filterable` decides only whether the companion is worth the narrower declared type. The arity was right; the reason given for it was not Also fixes the reference docs promising that nothing is dropped, which held only for an array edge, and counting two consequences before listing three.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #798.
A welded co-element filter – this agent in this role – hangs Typesense 30.2 indefinitely whenever the nested entry’s leaves hold arrays, which is every document a qualified relation produced in practice. Through the GraphQL surface that reached a consumer as
Unexpected error.What this changes, and why it is not a workaround
The hang is why this got noticed. It is not why the shape changes.
A weld asks whether one entry satisfies every condition. An entry holding
{ role: ["etser", "etcher"], creator_id: ["p1", "p3"] }stands for four (role, agent) pairs at once, so it answers the weld with none of them – and the weld silently degenerates into the cross-product it exists to exclude. That entry is not a valid input the engine mishandles; it is a shape that never had a meaning, and one ADR 24’s own words – one entry per edge – already exclude.So: a weldable nested leaf is single-valued, and multiplicity moves to the entry list.
searchSchemarefuses a nested field declaring bothfilterableandarray.output-only leaves are untouched – nothing welds them, so they may carry a list for display.transform, folding and the facet companion unchanged.locallookup is nested just as much as a Reference Type.filterable), and is declared under the arity of its path. That companion is the leaf a weld actually names, so it was the load-bearing half: fan-out alone still leftcreator_id: ["p1"]inside each entry, which is exactly what hangs 30.2.Full reasoning, measurements and the rejected alternatives are in the new ADR 26.
Measured
Against live containers, 1 000 000 documents, same data and same query:
The upgrade unwinds nothing: the fanned-out shape is no slower on 31.0 than the array shape it replaces, so when 31.0 goes stable this is routine maintenance rather than a migration back. And facets stay on the real fields: the other shape that works on 30.2 is an index-time composite
role|agentkey, faster at 3 ms but faceting into 2 142 125 buckets in 1.4 s against 15 buckets in 48 ms.Cost: indexing is ~15 % slower (54 s against 47 s per 1 000 000 documents) for ~50 % more entries. A batch cost, not a request cost.
No bound, on purpose
An earlier revision capped the entries a document stores (
maxEntries, default 100). It has been removed, because it guards the wrong end: by the time the projection runs the CONSTRUCT has matched those values, the endpoint has paid for it, they have crossed the wire, and framing has materialised them into one node. Capping the product declines the last and cheapest step while keeping every expensive one, and bounds nothing framing did not already hold. It also did nothing for the linear case that predates fan-out – a wide edge, or a display-only nesting of ten thousand entries.The bound belongs where the data enters memory. Capping values per leaf there makes the product
k^(weldable leaves), and leaf count is a schema constant – a bound in the schema’s own units rather than a number written against the data’s. That touches a seam serving every consumer of framing, so it is tracked separately, with its own evidence and its own record: #826.Until then the fan-out is unbounded, exactly as the entry list it replaces always was. What changed is that the growth can now be multiplicative rather than linear.
Defects found in review
Three review rounds found seven real defects, six introduced by this change and none visible from a unit test or the diff. Worth naming, since the fixes are most of the commit history:
typeat all for a locally-nested Root Type’s facet, which the engine refuses outright;facetablealone, which fan-out never splits, dropping ids silently;Every one was caught by running the projection or a live container, which is why
welded-filter.integration.test.tsasserts against a real engine rather than a compiled filter string.Notes for review
filterableandarray. Threesearch-pipelinefixtures carried exactly that shape and are updated here – the same edit a deployment makes.search-typesense(95.69 → 95.87), down 0.01 insearchwhere the cap’s tests were removed along with the code they covered.array: true.@lde/search-api-graphql:testis flaky, independently of this branch – Nx now flags it, and itsprint-sdlcase depends on module-graph state (vi.resetModules()then mockingprettier). It failed once here and passed on re-run.